-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Filebeat] Improve ASA/FTD Ingest Pipeline #23766
[Filebeat] Improve ASA/FTD Ingest Pipeline #23766
Conversation
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
Pinging @elastic/siem (Team:SIEM) |
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding some small comments
x-pack/filebeat/module/cisco/asa/test/additional_messages.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/cisco/asa/test/asa-fix.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/cisco/meraki/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/cisco/asa/test/additional_messages.log-expected.json
Outdated
Show resolved
Hide resolved
jenkins run the tests please |
Pinging @elastic/ingest-management (Team:Ingest Management) |
6bfca13
to
ae59310
Compare
jenkins, run tests |
Thanks for kicking that off @andrewkroh - 4 of the tests failed (dictionary item adds due to ICMP processing). I decided to simplify the PR by removing any changes to ICMP at all (will create a seperate PR for that). Could you kick Jenkins off again? |
jenkins, run tests |
I'm not sure why this build failed. Looks like something maybe in the backend with the Windows 2019 environment setup for the build? |
jenkins, run tests |
Thanks a lot @andrewkroh - looks like it passed and can be reviewed by the relevant teams. Appreciate the assistance. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your work on this. It LGTM, but it needs an entry in the changelog to describe the change. Can you please put a note into CHANGELOG.next.asciidoc at the root of the repo.
Added changelog entry. |
Looks like we have a conflict in fields.go due to the fields changes. This will need a rebase. |
0a80440
to
5bbd0c5
Compare
@andrewkroh - I have rebased and cleaned up some of the commits. LMK if something else needs work. Thanks! |
jenkins, run tests |
@andrewkroh if you could take a look, seems it's stuck in build or something odd. Thanks! |
jenkins, run tests |
@andrewkroh looks like the build is failing from when you pushed in the golden files commit. Looks like expected output for cisco AMP module changed, and that's failing the build. |
jenkins, run tests |
c090416
to
2d7997b
Compare
Fixes elastic#21658 For messages 716002: - Changed to GROK; allows for better parsing of event.reason - Added field for cisco.webvpn.group_name - Added field for event.reason per cisco docs for why session was terminated - Added field for cisco.termination_user for the AAA username terminating the connection For messages 722051: - Add angle brackets to dissect to properly dissect the message, per cisco docs - Added field for cisco.webvpn.group.name For messages 305011: - Change to GROK; allows for variance in message format with identity firewall For messages 302020: - Added GROK pattern to allows for variance in message format with identity firewall For messages 302014/302016/302021: - Added patterns and modified order of patterns of GROK to better match teardown messages - Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match. - Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future. - Added icmp_type and icmp_code parsing for 302021 messages - Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME. And: - Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor. - Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. - Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
2d7997b
to
0c52b2d
Compare
Fixes elastic#21658 For messages 716002: - Changed to GROK; allows for better parsing of event.reason - Added field for cisco.webvpn.group_name - Added field for event.reason per cisco docs for why session was terminated - Added field for cisco.termination_user for the AAA username terminating the connection For messages 722051: - Add angle brackets to dissect to properly dissect the message, per cisco docs - Added field for cisco.webvpn.group.name For messages 305011: - Change to GROK; allows for variance in message format with identity firewall For messages 302020: - Added GROK pattern to allows for variance in message format with identity firewall For messages 302014/302016/302021: - Added patterns and modified order of patterns of GROK to better match teardown messages - Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match. - Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future. - Added icmp_type and icmp_code parsing for 302021 messages - Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME. And: - Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor. - Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. - Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. (cherry picked from commit b5e43fc)
Fixes #21658 For messages 716002: - Changed to GROK; allows for better parsing of event.reason - Added field for cisco.webvpn.group_name - Added field for event.reason per cisco docs for why session was terminated - Added field for cisco.termination_user for the AAA username terminating the connection For messages 722051: - Add angle brackets to dissect to properly dissect the message, per cisco docs - Added field for cisco.webvpn.group.name For messages 305011: - Change to GROK; allows for variance in message format with identity firewall For messages 302020: - Added GROK pattern to allows for variance in message format with identity firewall For messages 302014/302016/302021: - Added patterns and modified order of patterns of GROK to better match teardown messages - Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match. - Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future. - Added icmp_type and icmp_code parsing for 302021 messages - Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME. And: - Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor. - Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. - Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. (cherry picked from commit b5e43fc) Co-authored-by: hitchatwork <56573831+hitchatwork@users.noreply.github.com>
What does this PR do?
Fixes #21658
For messages 716002:
For messages 722051:
For messages 305011:
For messages 302020:
For messages 302014/302016/302021:
And:
Why is it important?
Improved parsing of ASA/FTD messages mentioned above.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.How to test this PR locally
Related issues